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Abstract 

Assessing the effectiveness of educa- 
tional technology professional de- 
velopment (ETPD) must go beyond 
obtaining feedback from participants 
about their level of satisfaction with 
a workshop presenter. Effective and 
meaningful assessment of ETPD re- 
quires that we design inservice learning 
activities that can be measured using 
methods that are consistent with what 
we know about teaching and learning, 
recognize teacher and student change 
as it relates to the larger teaching and 
learning context, and view evalua- 
tion as an inseparable component of 
ongoing teacher action. We therefore 
offer for consideration an ETPD as- 
sessment model that merges three 
theoretical constructs through which 
professional development consum- 
ers might interpret research findings: 
(a) technological pedagogical content 
knowledge (TPACK), (b) organiza- 
tional learning, and (c) participant 
research and inquiry. (Keywords: pro- 
fessional development, assessment, 
TPACK, action research) 


W e’ve all been there. As you 

gather your notes and belong- 
ings at the end of a long day 
of learning some trendy, new technol- 
ogy tips and tricks, someone hands you 
a page with a few questions to answer: 
What did you learn today? How likely 
are you to use what you learned? Was the 
presenter effective? You’re anxious to get 
home, eyes weary from peering at the 
computer screen all afternoon, so you 
do your best to circle the numbers that 
would be complimentary to the present- 
er — she tried hard to be funny and keep 
our attention, after all — but you skip 


past the large, open-ended comments 
box in your rush to get out the door. 

Or maybe you are the professional 
developer charged with collecting data to 
evaluate the outcomes of a grant-funded 
project. You plan ahead with teachers and 
other school staff. You provide an intro- 
duction during large- and small-group 
activities. Later you proceed to “collect 
data” in classrooms, but instead you find 
yourself helping students and teachers 
with new software features and collabora- 
tive student projects. While in the thick 
of providing support to the success of 
classroom implementation, you might 
find yourself setting aside your observa- 
tion protocol. Soon the class period is 
over and students have no time to enter 
comments for the daily journal prompts 
you developed (those data you planned to 
use for the evaluation). 

These scenarios are all too com- 
mon, and yet these data — haphazardly 
solicited and carelessly offered — may 
represent the total information source 
we have for judging the effectiveness 
of our efforts in preparing teachers to 
teach with 21 s, -century skills and tools. 
Any conclusions we draw from these 
data are necessarily limited, skewed, 
and of questionable validity. In the first 
scenario, data are likely more represen- 
tative of how personable the presenter 
was, or how tired the participants were, 
than how well facilitated, well designed, 
or well matched to learners’ needs the 
learning experience was. In the second 
scenario, the presenter frets over lost 
opportunities for meaningful data col- 
lection, acknowledging that the data set 
will be smaller than desirable. 

Assessing the effectiveness of educa- 
tional technology professional devel- 
opment must go far beyond obtaining 
feedback from participants about their 


level of satisfaction with a workshop 
presenter. The “impact the professional 
development activity had on pedagogi- 
cal change or student learning” (Lawless 
& Pellegrino, 2007 p. 579) is of par- 
ticular importance to our field’s efforts 
to prepare teachers at all levels to use 
technology effectively. However, basing 
our evaluation on incomplete data leads 
us to an incomplete understanding and 
incomplete assumptions about the role 
of professional developers, teacher edu- 
cators, and teachers. 

The genesis of our thoughts on the 
assessment of professional develop- 
ment in educational technology was our 
work in co-editing the text Transforming 
Classroom Practice: Professional Develop- 
ment Strategies in Educational Technol- 
ogy (TCP) (Borthwick & Pierson, 2008). 
Through the solicitation of proposals 
and collaboration with chapter authors, 
what became clear to us in so many ways 
is the necessary connection between 
professional development models and 
the assessment strategies used in the 
evaluation of each model. 

Effective Evaluation of Educational 
Technology Professional Development 

The climate of accountability in the 
early 21 st century has heightened the 
awareness of stakeholders at all levels 
of the education system to the need 
for data. These many audiences for 
the findings of educational technology 
professional development (referred to as 
ETPD throughout this article) demand 
more than descriptive studies reporting 
isolated anecdotal narratives, even if 
those narratives share compelling stories 
of success. Clearly, a planned evaluation 
strategy, in place from the inception of 
the project, could assist professional 
developers in understanding the extent 
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to which ETPD is effective, rigorous, 
and systematic. 

Unfortunately, the field has not yet 
arrived at a level we can term effective 
by means of rigor and system. Instead, 

PD efforts are deemed effective based on 
teacher self-report and opinion — those 
data so easily collected on photocopied 
surveys and so unrealistically depended 
upon as meaningful facts. These means 
fall considerably short of demonstrating 
“whether professional development has 
changed practice and ultimately affected 
student learning” (Hirsch & Killion, 2009, 
p. 468) and have led to a literature base of 
well-intentioned descriptions of promis- 
ing, yet isolated, implementations and 
informal lessons learned from individual 
programs. This “show-and-tell” line 
of publication misses the definition of 
rigor with the absence of stated research 
questions, planned designs, and matched 
and multiple data collection. It hinders 
the flow of research into practice, leaving 
educators to wonder what PD workshops 
they should attend to improve their 
teaching or their students’ learning, as 
little or “virtually no information exists to 
help consumers of professional develop- 
ment” (Hill, 2009, p. 473). Further cor- 
roborating the troubling state of profes- 
sional development literature, a 2009 
review of published accounts of profes- 
sional development with respect to stu- 
dent learning outcomes found that “only 
nine of the original list of 1,343 studies 
met the standards of credible evidence 
set by the What Works Clearinghouse” 
(Guskey & Yoon, 2009, p. 496). 

Researchers have suggested frame- 
works to guide the assessment of ETPD. 
Desimone (2009) suggests that research- 
ers seek a consistent framework to 
enable PD to be “based on a strong theo- 
retical grounding” confirmed through 
multiple methods including case study, 
correlational, and experimental ap- 
proaches (p. 186). Lawless and Pellegri- 
no (2007) have recommended an evalu- 
ation schema that addresses (a) types 
of professional development, (b) unit of 
analysis, and (c) designs and methods. 
Much earlier, professional developers 
had been advised to make concerted 
efforts to systematically collect data on 


professional development in terms of the 
teacher and student outcomes (Gus- 
key, 2000). The persistent challenge for 
professional developers as consumers 
of evaluation research reports is sifting 
out effects on teaching and learning that 
can be attributed to technology use from 
those results that are the result of other 
initiatives. Along these lines, Pianta & 
Hamre (2009) assert that the value of 
“observational assessment of teachers for 
leveraging improvements in educational 
outcomes is that they can be directly 
related to the investigation and ex- 
perimentation of specific interventions 
aimed at improving teaching” (p. 115). 

Effective and meaningful assess- 
ment of ETPD requires that we design 
inservice learning activities that can be 
measured using methods that are consis- 
tent with what we know about teaching 
and learning; recognize teacher and 
student change as it relates to the larger 
teaching and learning context; and view 
evaluation as an inseparable component 
of ongoing teacher action. We therefore 
offer for consideration an ETPD assess- 
ment model that merges three theoreti- 
cal constructs currently enjoying much 
note and utility, through which profes- 
sional development consumers might 
interpret research findings: (a) techno- 
logical pedagogical content knowledge 
(TPACK); (b) organizational learning; 
and (c) participant research and inquiry. 

Evaluating PD According to TPACK: 

The What 

Professional development comes in 
all sizes and flavors, and to make an 
accurate assessment of the quality 
and impact of an activity, professional 
developers must consider the variety of 
ways teachers learn and the variety of 
variables that could affect teacher learn- 
ing. However, simply recognizing that 
there are differences among professional 
development attributes, and recognizing 
just how those attributes can be inter- 
related for effective technology use, are 
two very different things. Layering any 
examination of ETPD findings with 
the TPACK model (Mishra & Koehler, 
2006; Pierson, 2001) provides a helpful 
lens through which to view the process 


in light of current pedagogical thinking 
for 2 1 st - century learners and teach- 
ers. The fields of educational technol- 
ogy and teacher education have come 
to agreement around the concept of 
TPACK to describe the meaningful use 
of technology in teaching and learning. 
Derived from Shulman’s (1986) notion 
of teaching as the intersection of content 
knowledge and pedagogical knowledge, 
the definition of 2 1 st - century teaching 
also demands excellence in technological 
knowledge. True technology integration 
is said to be at the intersection of all 
three elements (Pierson, 2001). Further, 
the intersection of any two of the ele- 
ments defines worthwhile knowledge 
sets: technological content knowledge, or 
technologies used for specific content 
applications; and technological pedagogi- 
cal knowledge, or technology use for spe- 
cific pedagogical purposes. Evaluating 
the effectiveness of professional devel- 
opment, then, must consider how well 
teachers are prepared to meaningfully 
use technologies in discipline-specific 
ways as well as ways that are compat- 
ible with multiple teaching and learning 
strategies and roles. 

It stands to reason that if the elusive 
goal of effective technology-integrated 
teaching can be found at the intersection 
of content, pedagogy, and technology, 
then it logically follows that at this same 
center will be effective assessment (see 
Figure 1, p. 128). Assessment is an inte- 
gral and inseparable part of the curricu- 
lum development and teaching process — 
one leading to the next and cycling 
back again. And if effective technology 
integration — and assessment of such — is 
the goal of educational technology profes- 
sional development, then these elements 
should be prominent in any evaluation 
model. Lawless and Pellegrino hint at this 
importance when they say that “the most 
important impact a professional develop- 
ment activity can have on a teacher is that 
of pedagogical practice change ostensibly 
reflecting a deeper change in pedagogical 
content knowledge” (p. 597). Likewise, 
Guskey and Yoon (2009) found that PD 
projects are effective when they focus on 
enhancing teachers’ content knowledge 
and pedagogical content knowledge. 
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Figure 7. Effective evaluation of professional development is at the center of the TPACK model. 


So, as the field continues to ex- 
plore the usefulness of the TPACK 
construct to define teacher knowledge 
with technology, it must also push that 
exploration into how TPACK can shape 
evaluation of ETPD efforts. 

Evaluating Professional Development 
within the Context of Organizational 
Learning: The Where 

The problem with those descriptive 
studies of isolated pockets of success- 
ful professional development is that the 
authors are not always clear enough — and 
the readers make incomplete assump- 
tions about — the surrounding context 
of school, student, and administrative 
factors. Of course, a highly successful ef- 
fort in one instance may not work as well, 
even if implemented to the letter, in a less 
supported or funded or engaged con- 
text. In short, context matters, for both 
ETPD implementation and assessment. 

To assess the effectiveness of professional 
development in leading teachers to long- 
lasting gains in knowledge, attitudes, and 
instructional behaviors, we must examine 
supporting factors within the teaching 
and learning context. 

Professional development has been 
characterized in the literature as a 


variety of leveled developmental models 
with fixed sequences of stages and 
levels of knowledge and skills acquisi- 
tion (DalFAlba & Sandberg, 2006). 

Early frameworks for understanding 
how teachers learned to use technology 
aimed to shift the focus from the tech- 
nology tools themselves to the teachers 
and their developmental needs, and in 
doing so, they addressed the uniqueness 
of learners who participate in any pro- 
fessional development session. The role 
of the school organization, then, was to 
address these individual needs, moving 
teachers to more advanced stages. 

Elowever, although assessing teachers’ 
needs, as well as understanding the types 
of assistance teachers might require 
and concerns they might have based on 
developmental levels, are indeed neces- 
sary steps, the focus on a single teacher’s 
needs and progress cannot reveal all 
requirements for success. In fact, the 
implied linear nature of these staged 
models may conceal “more fundamental 
aspects of professional skill develop- 
ment” (DalFAlba & Sandberg, 2006, p. 
383). Rather, we need to examine the 
learning of an individual within the 
learning of the system as a whole; this 
implies a shift toward thinking about 


how individuals fit into the larger orga- 
nization and the additional learning that 
must take place on the organizational 
levels. Successful educational technology 
PD initiatives are characterized by an ex- 
panded, informed, and connected view 
of learning on both the individual and 
the organizational level. 

A systems view suggests a focus not 
only on individual teacher and student 
growth, but also on changes in organiza- 
tion policies and procedures, infrastruc- 
ture, curriculum and instruction, expecta- 
tions for stakeholders, and organizational 
climate (Newman, 2008). Teacher growth 
and change will not be sustained without 
organizational support and, in fact, may be 
sabotaged (Borthwick & Risberg, 2008). 
Harris (2007) recommends that making 
choices about the particular professional 
development methods and strategies — 
selecting, combining, and sequencing 
aspects to craft an overall approach based 
on its unique needs as a learning organi- 
zation — is exactly what a school or district 
must do. Trying out new technologies 
requires teachers to assume a level of risk- 
taking; to allow for success, teachers will 
need to work in “a climate of trust, col- 
laboration, and professionalism” where ad- 
ministrators “promote technology-related 
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risk taking among teachers on behalf of 
students” (Borthwick & Risberg, 2008, 
p. 39). Desimone (2009) sees context as 
“an important mediator and modera- 
tor” for implementation of professional 
development models (p. 185), and even 
further, Zhao, Frank, and Ellefson (2006) 
suggest, in correlating classroom technol- 
ogy use with professional development 
activities, that “schools should develop 
a culture instead of a program of profes- 
sional development” (p. 173). Evaluation 
of ETPD effectiveness must, then, include 
an assessment of the context in which 
that development is occurring. In that 
way, consumers of ETPD findings can 
determine in what ways they can scale a 
successful project, locally situated within 
the context of a school whose staff are 
committed to continuous learning, to dif- 
ferent contexts. 

Evaluating Professional Development 
through Practitioner Research: The How 

Change in pedagogical practice is the ul- 
timate goal of professional development. 
However, few studies use data on teacher 
use to inform their practice (Lawless 
& Pellegrino, 2007). So, even if profes- 
sional developers ensure that assessment 
of ETPD focuses on teacher knowledge 
(TPACK) and occurs in a supportive 
context, how can they feed the results 
back into practice and allow that prac- 
tice to inform continued research? The 
simple answer is that the two — research 
and practice — must in fact be one and 
the same. 

The evaluation of such a complex 
educational endeavor as ETPD must em- 
ploy multiple, flexible, systematic, and 
rigorous methods. It is the latter two on 
this list that are infrequently found in re- 
ports of ETPD. Yet, if approached from 
a rigorous stance, even such research 
methods as action research and case 
study can meet the “platinum standard” 
of research. Editors of journals in tech- 
nology and teacher education propose: 
“The platinum standard requires rigor- 
ous research in authentic school settings 
that approaches idealized designs as 
nearly as possible, given the constraints 
of schools and real-world learning envi- 
ronments” (Schrum et al., 2005, p. 204). 


These editors recommended undertak- 
ing research based on authentic class- 
room settings as long as it is grounded 
in theory and builds upon the existing 
knowledge base. Operating within a 
platinum research standard opens up as 
“acceptable” those forms of experimental 
research that do not require a control 
group. The approach is a more reason- 
able methodological goal for classroom- 
based research than a true experimental 
model, which is often referred to as the 
“gold standard.” 

In particular, action research, with its 
practical and indigenous classroom ap- 
plications (Nolen & Putten, 2007, p. 403) 
can provide a framework to explicitly 
connect professional development to 
evaluation methodology. As explained 
by Ham, Wenmoth, & Davey (2008), 
a self-study approach, then, might not 
only be the professional development 
method, but also serve as a form of 
assessment, leading to larger, common 
answers about student outcomes. Such a 
spiraled sequence of inquiry, data collec- 
tion and analysis, and reflection uses the 
results of systematic inquiry to inform 
and lead into the next phase of question- 
ing. The role of teachers as participant 
researchers is critical to the diagnosis 
of learning outcomes, identification of 
subsequent instructional strategies, and 
input to policy development (Borthwick, 
2007-2008). Participant research as a 
2 1 st - century professional development 
assessment model allows teachers as 
professionals to look at their practice 
in new ways (Linn, 2006); respects 
teacher knowledge and experience; and 
provides a long-term as well as immedi- 
ate evaluation and feedback loop, with 
small findings continuously driving the 
next steps of instruction (Fullan, Hill, & 
Crevola, 2006). 

In other words, instead of just outside 
evaluators coming in to assess teach- 
ing practice at the end of a prescribed 
period and some time later feeding 
those findings back into another profes- 
sional development workshop, teachers 
as researchers constantly ask questions 
of their teaching, collect and analyze 
multiple forms of data, collaborate with 
one another, and feed ongoing findings 


into tomorrows teaching plans. This 
metacognitive approach to the evalua- 
tion of professional development enables 
teachers’ lifelong learning, thus extend- 
ing the reach of every formal profes- 
sional development effort. 

Unfortunately, in actual practice, 
classroom teachers are rarely supported 
or encouraged to engage in research or 
scholarly writing; thus, this metacogni- 
tive approach may not ever come to be. 
Ideal partners to facilitate this process 
are those in higher education — re- 
searchers in educational technology and 
related fields who are regularly engaged 
in such scholarly activity (Cunningham, 
et al, 2008). Teachers who have experi- 
enced the opportunity to collaborate and 
co-teach with such outside technology 
partners were afforded opportunities 
to experiment with teaching with new 
technologies (Zhao, Frank, & Ellefson, 
2006). Such school- university part- 
nerships can create the framework for 
ongoing co-research habits that will 
continually inform classroom practice 
and research alike. 

A Model for Effective, Rigorous, and 
Systematic Evaluation of Educational 
Technology Professional Development 

Assessment of ETPD must recognize the 
interaction among the TPACK variables 
of content, pedagogy, and technology 
to understand the extent of meaningful 
technology-supported teaching. Assess- 
ment of ETPD must consider the teach- 
ing and learning context, both locally 
and broadly. And assessment of ETPD 
must close an action research loop join- 
ing research with practice. We propose 
extending the TPACK model, defining 
effective technology- enhanced teaching, 
to guide the assessment of ETPD when 
supported by the surrounding “frame” 
of context and participant research, 
specifically phases of Reflection, Inquiry, 
Collaboration, and Sharing (see Figure 
2, p. 130). In this model, individual 
and organizational learning occur over 
and over again as educators and their 
research partners engage in the action 
research process in light of the organiza- 
tional context, thus positioning assess- 
ment of ETPD as a culture of learning. 
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Figure 2. Assessment of ETPD as a culture of learning based on a contextually-situated and inquiry-framed TPACK model. 


This contextually situated and inqui- 
ry-framed model of ETPD positions 
TPACK as not only the center of a con- 
ceptual model for effective professional 
development, but also as the center of 
a comprehensive plan for assessment 
embedded in the work and culture of 
teaching. Further, it implies that suc- 
cessful assessment of PD efforts must 
have at their centers the basic defini- 
tional components of TPACK elements. 
They must resemble the iterative action 
research cycle: encouraging PD facili- 
tators to begin any staff development 
session by asking participants to think 
about what they can learn from it and 
how what they will learn is situated in 
the work that they already do, by posing 
questions about how teaching and learn- 
ing can improve, by collaborating with 
peers and more experienced colleagues 
to solve problems of practice, and by 
evaluating and sharing findings with 
one another as part of an ongoing effort 
at collective improvement. Such a cycle 


of inquiry exemplifies Fullan, Hill, and 
Crevolas (2006) Breakthrough model, in 
which “teachers will operate as interac- 
tive expert learners all the time” (p. 95). 
Observing and documenting teacher 
behaviors “in the thick” of classroom 
practice, whether done by teachers as 
participant researchers or in collabora- 
tion with research partners, suggests the 
most direct route to evaluating effects of 
both formally supported and personally 
initiated professional development. 

The Utility of this Model 

The potential power of ETPD to en- 
hance teacher knowledge and skills, and 
thus improve student learning, means 
it is worth our time to understand what 
works and in what contexts. Although 
there are some good, reliable tools 
available, “they have not been subject 
to repeated use and validation and are 
not widely available” (Desimone, 2009, 
p. 192). Collaborative research efforts 
such as the Distributed Collaborative 


Research Model (DCRM) (Pierson, 
Shepherd, & Eeneway, 2009) show the 
promise of providing guidance about 
how assessment of ETPD might meet 
the “platinum standard” of educational 
technology research. The intentionally 
collaborative nature of DCRM embeds 
into the research design from the outset 
the capacity for rigorous and valid prac- 
tice. Partnerships, both in the form of 
higher education partners working with 
school partners and partners collaborat- 
ing across distant locations to obtain a 
larger database about which to speak 
more broadly regarding joint findings, 
are at the heart of this planned col- 
laborative research. The DCRM model 
further recommends the need for con- 
sistent methods and data collection tools 
to facilitate interinstitutional collabora- 
tions. Assessment tools with which we 
are familiar, including observations, 
surveys, interviews, and text and video 
analysis, all have value. 

The quest for rigor will not be easy. 
Chapter authors in our TCP text spent 
much more time describing the activities 
undertaken as part of the implementation 
process of their model than providing 
evidence of student achievement. There 
are several logical explanations for this, 
including minimal budgets for program 
evaluation and minimal time for par- 
ticipant observers to collect data when 
they were in the thick of preparing for 
and implementing professional develop- 
ment activities. But our expectations as 
consumers of professional development 
reports need to change. We must expect 
a reporting of evaluation measures, 
something that now is rarely done (Desi- 
mone, 2009). We must expect that we 
see detail about “what works, for whom, 
and under what conditions” (Lawless & 
Pellegrino, 2007, p. 599). We must expect 
that the ETPD community will share 
findings among themselves, such as in a 
clearinghouse (Nolen & Putten, 2007) or 
in database format, such as the Action 
Research for Technology Integration 
(ARTI) (http://etc.usf.edu/fde/arti.php) 
at the University of Florida (Dawson, 
Cavanaugh, & Ritzhaupt, 2008). 

Instead of viewing the complexify 
of classroom implementation of new 
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instructional approaches as a detriment, 
we must learn to base ETPD in the heart 
of where the action takes place. And 
in fact our proposal of a contextually 
situated and inquiry-framed TPACK 
model as a template for the design of 
ETPD capitalizes on the work of teach- 
ers and research partners who embrace 
an inquiry approach to teaching and 
learning, connecting systematic evalua- 
tion of their participation in professional 
development activities directly to their 
effectiveness in the classroom. No matter 
what format PD takes, through all types 
of partnered approaches — including 
mentoring, peer coaching, students as 
professional developers, and professional 
learning circles — a contextually situated 
and inquiry- framed model of ETPD 
assessment can scaffold the assessment 
process. If we seek validity in our work 
and reporting, we must commit ourselves 
to systematic study of our work and 
documentation of related outcomes. In 
this way, our expectations as consumers 
of— and participants in — professional 
development evaluation can change. 
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